It is very important in the inspection of digital image quality to determine whether there is a defective region where pixel data changes non-uniformly. Especially, recent years have seen demand for image sensor devices growing in increasing numbers of applications, such as digital still cameras and mobile phones with a built-in camera. There is a large demand for image sensor devices with enhanced quality, accordingly. To achieve this, spot defects and stain defects, which are defective regions in a digital image (image sensor image) output of an image sensor device, increasingly need to be detected in the course of an inspection step.
Image data is never uniform across the screen even on a high quality image sensor device due to shading and noise. The “shading” refers to gradual changes in pixel data (pixel values) with respect to the coordinates of a pixel. Especially, when an image sensor image has a point defect, spot defect, or stain defect, the pixel data changes in a complex manner in accordance with that particular defect. This is the primary reason why spot defects and stain defects needs to be inspected.
The image sensor devices have increasing numbers of pixels, while delivery time for products with short product life cycles, like mobile phones, is becoming shorter. The situation has created another strong demand for reduced inspection time.
Conventionally, the image sensor image were inspected visually by inspection personnel for spot defects and stain defects. The inspection inevitably relies on the subjective evaluation by the inspection personnel. This could be a cause for different inspection standards among inspection personnel, inconsistent inspection results due to the physical condition of the inspector at the time of inspection, and difficulty in quantification of spot and stain defects. Inspection apparatus has been recently developed which quantifies and detects spot and stain defects. Automated processes are gradually being introduced.
Now, we define a “spot defect” and a “stain defect.” A “point defect” which occurs in an image sensor image is also defined.
A “point defect” is a pixel, in an image sensor image, where data assumes vastly and markedly different values when compared with the surrounding pixels. A “spot defect” is a cluster of pixels where data assumes different values from the surrounding pixels. The differences are however smaller than those in the case of a point defect. A “stain defect” is a larger cluster of pixels than a spot defect where data assumes different values from the surrounding pixels. The differences are however smaller than those in the case of a spot defect.
FIG. 21 is a schematic showing a point defect, spot defect, and stain defect. Referring to the figure, in the inspection image 100, the point defect 110 occupies a small area. The spot defect 120 occupies a greater area. The stain defect 130 occupies an even greater area. The “inspection image” is a sample image output of an image sensor for the purpose of detecting spot defects and stain defects.
Next, we will describe shading. “Shading” refers to a phenomenon in which pixel data changes gradually over a greater region than the stain defect with the pixel data decreasing toward the top/bottom/right/left edges of the image sensor image. This is due to relatively poor sensitivity of pixels along the edges of the image in comparison to the sensitivity of the pixels at the center.
FIG. 22 shows an inspection image 100, as an example, in which a spot defect 120 occurs. In FIG. 22, shading 140 is observed in which pixel data decreases toward the top/bottom/right/left edges of the inspection image 100.
As mentioned earlier, inspection apparatus has been recently developed which quantifies and detects spot and stain defects, and automated processes are gradually being introduced. As an example of such technology, we refer to a non-patent document, “Image Quality Inspection Algorithm for Flat Panel Displays,” Yokokawa Technical Report, Vol. 47, No. 3, 2003.
As shown in a flow chart in FIG. 23, the technology described in the document involves two major processes: a defect enhancement process (step 100, or simply “S100”) and a defective region extraction process (S200). The image inspected for spot defects and stain defects by the flow in FIG. 23 is produced by shining white or other monochromatic light which shows no changes in luminance or color onto an image sensor device.
The defect enhancement process (S100) filters out shading, noise, and point defects to render spot defects and stain defects in the inspection image easier to detect. Specifically, the defect enhancement process, as shown in FIG. 23, includes an image reduction step (S110), a shading correction step (S120), a noise removing step (S130) with a smoothing filter, a point defect removing step (S140) with a point defect removing filter, and an edge detecting step (S150) with a Laplacian filter. These steps in the defect enhancement process (S100) will be described later in detail.
The defective region extraction process (S200) includes a step for binarizing pixel data (S210), a step for removing concave/convex noise from the inspection image binarized in S210 (S220), a step for allocating identification numbers to potentially defective regions (S230, or “labeling step), a step for calculating feature quantities (spot area, contrast, luminance gradient, etc.) of all the regions identified by numbers in S230 (S240), and a step for determining from the feature quantities calculated in S240 whether the potentially defective regions are really defective (S250). These steps in the defective region extraction process (S200) will be described later in detail.
The steps in the defect enhancement process (S100) and the defective region extraction process (S200) will now be described one by one.
The image reduction step (S110) reduces the image in both vertical and horizontal dimensions. The downscaling reduces the data size of the inspection image and makes spot and stain defects more prominent.
The shading correction step (S120) compensates for reduction in pixel data due to shading to remove the effect of shading from the inspection image. The shading correction step, for example, calculates average pixel data for 9 pixels in a 3×3 pixel matrix with a target pixel at its center and replaces the pixel data of the target pixel with that pixel data divided by the average pixel data.
The noise removing step (S130) removes noise from the inspection image with, for example, a smoothing filter. The smoothing filter refers to a filter capable of removing high frequency components from the image, thus removing noise. FIG. 24(a) shows an example of this filtering, where the smoothing filter has calculated average pixel data for the 9 pixels in a 3×3 pixel region with a target pixel at its center and replaced the pixel data of the target pixel with the average pixel data.
The point defect removing step (S140) removes point defects from the inspection image with a point defect removing filter. The point defect removing filter refers to a filter which, for example, calculates the median, maximum, and minimum pixel data for the 9 pixels in a 3×3 pixel region with a target pixel at its center and if the pixel data of the target pixel is either the maximum or minimum, replaces that pixel data with the median pixel data.
The edge detecting step (S150) enhances the edges of spot defects and stain defects with, for example, a Laplacian filter. The Laplacian filter refers to a filter extracting edges in density change from the image. FIG. 24(b) shows this filtering, where the Laplacian filter has subtracted the pixel data of the top/bottom/right/left pixels of a target pixel (4 neighboring pixels) from four times the pixel data of the target pixel and replaces the pixel data of the target pixel with the difference.
FIG. 25 shows a formula embodied in the filters in FIG. 24(a) and FIG. 24(b). Generally, in filtering, it should be considered which parts of the image are eligible for the filtering. For example, filtering on 3×3 pixels does not work with pixels along the edges of the image, since those pixels do not have a complete set of adjacent pixels. So, attention should be paid to the initial and last values in an image scan loop.
The binarizing step (S210) compares pixel data of the inspection image with a predetermined threshold to convert the data to binary data (0s or 1s). The binarization is aimed at reducing the quantity of information and facilitating the extraction of geometric shapes and the calculation of their areas. The threshold by which the binary data is determined to be either 1s or 0s may be a constant value across the image or specified for each region.
The inspection image binarized in the binarization step S210 is then subjected to the noise removing step S220 where concave/convex noise is removed. The noise removing step is done by dilation/erosion, isolated point removal, etc.
Dilation/erosion will be now described. Dilation/erosion is an image processing method which removes fine concave/convex noise around a targeted object from a binary image. Specifically, in a dilation, if even only one of the pixels in the proximity of a target pixel (close pixels) is “1,” the pixel data of the target pixel is set to 1. In an erosion, if even only one of the pixels in the close pixels is “0,” the pixel data of the target pixel is set to 0. Dilation and erosion are often used together. A dilation, followed by an erosion, removes small concave noise from the object. An erosion, followed by a dilation, removes small convex noise.
Referring now to FIG. 26, the process from the binarized inspection image 150 to the dilation/erosion noise removal will be described. Assume that the inspection image 150 has three noise pixels 151. The noise pixels 151 are removed in a dilation carried out on the inspection image 152 which is a result of an erosion on the inspection image 150. The result is an inspection image 153. In FIG. 26, “1” pixels are white squares, and “0” pixels are black squares.
The isolated point removal, for example, replaces the pixel data of a target pixel with the pixel data of its 8 surrounding pixels in a binary image if the pixel data of the target pixel is 0 (or 1) and the pixel data of its 8 surrounding pixels is 1 (or 0).
Next, the labeling step S230 will be described. The labeling step divides continuous objects in the image into groups and allocates identification numbers. Normally, the labeling step, while scanning the whole image, determines whether or not pixels are continuous and allocates labeling numbers to the pixels. In the labeling step, those objects which are determined to belong to different groups in an initial stage of the scan may turn out, after the scan, to belong to a single continuous object. The labeling should therefore be done sequentially in normal situations.
Suppose, as an example, an inspection image 160 with pixel data specified as shown in FIG. 27. The inspection image 160 includes three continuous objects 161, 162, 163 where pixel data is all 1s. Carried out on the inspection image 160, the labeling step allocates identification numbers a, b, c to the objects 161, 162, 163 respectively as shown in FIG. 27.
After the labeling step, feature quantities (spot area, contrast, luminance gradient, etc.) are calculated for all the numbered pixel data (S240). Based on the feature quantities calculated in S240, it is determined whether the potentially defective regions are really defective (S250).
A labeling step performed on the inspection image shown in FIG. 27, as an example, designates the objects 161, 162, 163 potentially defective regions. In S240, feature quantities are calculated for each of the potentially defective regions. For example, the spot area, one of the feature quantities, is obtained by counting pixels where pixel data is 1.
With growing numbers of pixels in an image sensor image, the filtering steps and the identification number allocating step, required to enhance spot defects and stain defects in the conventional art described in the non-patent document, take an increasingly significant amount of time. This in turn means that the image sensor device needs an ever longer period of time for testing.
Furthermore, an original or scaled-down image is needed to calculate contrast, luminance gradient, and other feature quantities to determine whether the potentially defective regions are really defective. These images require memory where they are stored, which adds to circuit complexity.
Circuit complexity increases also if the conventional image processing method is implemented in hardware, for example, in the DSP section of the image sensor device or module or in a dedicated image processing device, because the conventional art described in the non-patent document involves repeated filtering to detect spot defects and stain defects.
These two problems (1. extended test time on the image sensor device; 2. increased circuit complexity) occur not only when the conventional art described in the non-patent document is applied to the inspection of an image sensor image. The same problems also occur when the art is applied to the inspection of a general digital image for defective regions. The conventional art described in the non-patent document, if used to inspect a general digital image for any defective regions, will face the two problems: 1. It takes an extended period of time to determine if the image contains any defective regions. 2. Circuit complexity increases.